Image processing device, image processing method, and program recording medium

ABSTRACT

The invention enables image processing of visible light and near-infrared light using an imaging device. An acquisition unit ( 110 ) acquires an image signal representing an image including near-infrared light that has an intensity according to a pattern having a prescribed geometric shape. A signal processing unit ( 120 ) uses pattern information which defines the pattern to output, a color signal representing visible light components corresponding to the image signal and a near-infrared signal representing near-infrared light components corresponding to the image signal.

TECHNICAL FIELD

The present invention relates to video processing.

BACKGROUND ART

Color image input devices, such as a digital still camera and a videocamera, generally have a configuration in which an optical filter ofthree colors of red (R), green (G), and blue (B) is incorporated in animage sensor. Light incident on such a color image input device isdecomposed by the three-color optical filter and converted into signalscorresponding to R, G, and B colors, respectively, by the image sensor.

When a silicon-based sensor is used as the image sensor used for thecolor image input device, sensitivity of the sensor ranges from avisible range to a near-infrared range. However, a near-infrared lightcomponent may have an adverse effect on color reproduction. However, thethree-color optical filter guarantees a constant transmittance for awavelength range corresponding to each color, but does not necessarilyguarantee an optical transmission characteristic for ranges other thanthe visible range, such as the near-infrared range.

FIG. 17 is a diagram exemplifying a spectral transmittance of an RGBthree-color optical filter. For example, when the visible range is setfrom 400 to 700 nm, it is expected that the filters for the respectivecolors have characteristics for transmitting light having a wavelengthin the vicinity of 400 to 500 nm (B), 500 to 600 nm (G), and 600 to 700nm (R). However, the filters for the respective colors may also havecharacteristics for transmitting light in a range other than the visiblerange as illustrated in FIG. 17.

Spectral sensitivity characteristics of an image sensor using aphotodiode that is often adopted in an image input device includesensitivity in a wavelength range of 700 nm or more. In this case, whenonly applying a three-color optical filter having spectral sensitivitycharacteristics as illustrated in FIG. 17 to a typical image sensor,this may cause a problem in terms of color reproducibility. Accordingly,when the image sensor is required to have high color reproducibility,the image sensor is provided with an infrared cut filter.

FIG. 18 is a diagram illustrating color-matching functions of an XYZcolorimetric system associated with human color perception. Asillustrated in FIG. 18, the human color perception does not includesensitivity for light in a wavelength of 700 nm or more. Accordingly,light having power in a wavelength range of 700 nm or more does notaffect a perceived color that is a psychophysical value.

Herein, a case is assumed where light having power in a wavelength rangeof 600 nm or more as illustrated in FIG. 19 is observed. This light isperceived as red by a human. On the other hand, when the light isobserved by an image sensor using a three-color optical filter asillustrated in FIG. 17, the output signal includes not only an R value,but also G and B values. Accordingly, the output signal represents acolor different from the color (red) perceived by a human.

In the color image input device, an infrared cut filter having aspectral transmittance for removing an effect of near-infrared light ina wavelength of 700 nm or more as illustrated in FIG. 20 is used inorder to implement color reproducibility according to the human colorperception. Specifically, as illustrated in FIG. 21, the optical systemof the color image input device is provided with an infrared cut filter610 to thereby block incidence of near-infrared light on a three-coloroptical filter 620 and an image sensor 630. This configuration allowslight having no power in the near-infrared range to be incident on thethree-color optical filter 620 and the image sensor 630.

On the other hand, in a case of capturing a video under a circumstancein which an amount of light is insufficient, high-sensitivity capturingin which noise is suppressed is required. In such a case, it isdesirable to increase an amount of received light in the image sensor inorder that sensor noise due to an insufficient amount of light issuppressed. As a method for implementing high-sensitivity capturing in adark place, a capturing method using near-infrared light is known.

A simplest method using near-infrared light during high-sensitivitycapturing is a method in which an infrared cut filter set in an opticalsystem is mechanically moved during high-sensitivity capturing tothereby temporarily remove the infrared cut filter from the opticalsystem. However, this method has problems of an increase in the numberof components, i.e., an increase in cost, as well as an increase inpossibility of occurrence of a failure due to requirement for amechanical operation for moving the infrared cut filter.

On the other hand, NPL 1 discloses a method for capturing withoutrequiring any mechanical operation. Specifically, NPL 1 describes acapturing method using two cameras for capturing a color image and anear-infrared image, respectively.

Further, NPL 2 discloses, as illustrated in FIG. 22, an image sensor 700having a configuration in which a four-color optical filter obtained byadding an infrared (IR) filter for transmitting near-infrared light toan RGB three-color optical filter is incorporated. FIG. 2 of NPL 2illustrates spectral sensitivity characteristics of respective opticalfilters for R, G, B, and IR. The spectral sensitivity characteristics ofrespective optical filters for R, G, and B include spectral sensitivitysimilar to that of the IR filter in a near-infrared range. In order toimplement high color reproducibility during capturing in the daytime, itis necessary to suppress or eliminate an effect of near-infrared lightincluded in R, G, and B color signals. The image sensor described in NPL2 removes IR components included in the R, G, and B color signals duringcapturing in the daytime, and uses not only an IR signal obtained bycausing light to transmit an IR filter, but also IR components includedin the R, G, and B color signals during capturing at night, therebyobtaining a black-and-white image.

PTL 1 discloses an imaging device that generates signals of R, G, B, andnear-infrared (NIR)colors by using an R, G, and B three-color opticalfilter for transmitting NIR light, and a photosensor for detectingnear-infrared light. This photo sensor includes a visible light sensorunit at a shallow position in a light incident direction, and alsoincludes a non-visible light sensor unit at a deep position in thedirection.

In addition, NPL 3 discloses a method for generating a four-channelimage by separating a color channel and an NIR channel from an imagecaptured by using a color filter array different from a typical one, byusing two types of filters having different spectral transmissioncharacteristics for G filters of an RGB Bayer type color filter array(CFA), without using an IR cut filter, and the like.

Citation List Patent Literature

[PTL 1] Japanese Unexamined Patent Application Publication No.2011-243862

Non Patent Literature

[NPL 1] Sosuke Matsui, Mihoko Shimano, Takahiro Okabe, Yoichi

Sato, “Image Enhancement of Low-Light Scenes with Combination of ColorImage and Near Infrared Images”, The 12th Meeting on Image Recognitionand Understanding (MIRU 2009), collection of papers, pp. 1089-1096, 2009

[NPL 2] Shinzo Kayama, Keisuke Tanaka, Yutaka Hirose, “Day-and-nightimager for security monitoring cameras”, Panasonic Technical Journal,Vol. 54, No. 4, pp. 18-23, January, 2009

[NPL 3] Z. Sadeghipoor et al, “A Novel Compressive Sensing Approach toSimultaneously Acquire Color and Near-Infrared Images on a SingleSensor”, Proc. of IEEE ICASSP, pp. 1646-1650, 2013.

[NPL 4] O. Losson, L. Macaire, Y. Yang, “Comparison of Color DemosaicingMethods”, Advances in Imaging and Electron Physics, Vol. 162, pp.173-265, 2010.

[NPL 5] R. Ramanath, W. Snyder, G. Bilbro, W. Sander, “Demosaickingmethods for Bayer color array”, J. Electronic Imaging, Vol. 11, No. 3,pp. 306-315, 2002.

[NPL 6] S. Ferradans, M. Bertalmio, V. Caselles, “Geometry-BasedDemosaicking”, IEEE Trans. on Image Processing, Vol. 18, No. 3, pp.665-670, 2009.

SUMMARY OF INVENTION Technical Problem

The method described in NPL 1 is to generate an RGB image and anear-infrared image by using two cameras. Although the method describedin NPL 1 can be configured using one device, the method requires twooptical paths and two image sensors for the RGB image and thenear-infrared image, respectively.

The image sensors described in NPL 2 and PTL 1 are special image sensorsfor generating near-infrared images, and thus it can be said that theimage sensors do not have a typical configuration. The color filterarray described in NPL 3 requires two different G filters.

An object of the present invention is to provide a technique thatenables video processing of visible light and near-infrared light byusing a capturing device having a typical configuration.

Solution to Problem

An aspect of the invention is an image processing device. The imageprocessing device includes an acquisition means configured to acquire avideo signal representing a video including near-infrared light havingan intensity corresponding to a pattern having a prescribed geometricshape. And the image processing device includes a signal processingmeans configured to output a color signal and a near-infrared signal byusing pattern information for defining the pattern. The color signalrepresents a visible light component corresponding to the acquired videosignal. The near-infrared signal represents a near-infrared lightcomponent corresponding to the video signal.

Another aspect of the invention is an imaging device. The imaging deviceincludes a light receiving means including an optical filter configuredto transmit near-infrared light with a pattern having a prescribedgeometric shape, and being configured to generate a video signalrepresenting a video including near-infrared light transmitted throughthe optical filter. The imaging device includes an image processingmeans configured to output a color signal and a near-infrared signal byusing pattern information for defining the pattern. The color signalrepresents a visible light component corresponding to the generatedvideo signal. The near-infrared signal represents a near-infrared lightcomponent corresponding to the video signal.

Another aspect of the invention is an image processing method. The imageprocessing method includes acquiring a video signal representing a videoincluding near-infrared light having an intensity corresponding to apattern having a prescribed geometric shape; and outputting, by usingpattern information for defining the pattern, a color signalrepresenting a visible light component corresponding to the acquiredvideo signal and a near-infrared signal representing a near-infraredlight component corresponding to the video signal.

Another aspect of the invention is a computer-readable program recordingmedium recording a program. The program causes a computer to executeprocessing of acquiring a video signal representing a video includingnear-infrared light having an intensity corresponding to a patternhaving a prescribed geometric shape. And the program causes a computerto execute processing of outputting, by using pattern information fordefining the pattern, a color signal representing a visible lightcomponent corresponding to the acquired video signal and a near-infraredsignal representing a near-infrared light component corresponding to thevideo signal.

Advantageous Effects of Invention

According to the present invention, it is possible to perform videoprocessing of visible light and near-infrared light using a capturingdevice having a typical configuration.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a configuration of a videoprocessing device.

FIG. 2 is a diagram illustrating an NIR cut filter.

FIG. 3 is a block diagram illustrating a configuration of a videoprocessing device.

FIG. 4 is a block diagram illustrating a configuration of a second colorsignal estimation unit.

FIG. 5 is a flowchart schematically illustrating processing executed bythe video processing device.

FIG. 6 is a diagram illustrating a near-infrared signal.

FIG. 7A is a diagram illustrating near-infrared light corresponding to apattern of an NIR cut filter.

FIG. 7B is a diagram illustrating near-infrared light corresponding to apattern of an NIR cut filter.

FIG. 8 is a diagram for explaining values in Formula (2).

FIG. 9 is a diagram illustrating a relationship between an intensity ofnear-infrared light and a distance from the center.

FIG. 10 is a schematic view illustrating a configuration of a capturingdevice.

FIG. 11 is a schematic view illustrating a behavior of near-infraredlight incident on a light receiving unit.

FIG. 12 is a diagram illustrating a configuration of a color filter.

FIG. 13 is a diagram for explaining an example of demosaicingprocessing.

FIG. 14 is a diagram illustrating a correlation relation between aninfrared transmissive portion of an NIR cut filter and a color filter.

FIG. 15 is a schematic view illustrating a configuration of a capturingdevice.

FIG. 16 is a schematic view illustrating another configuration of thecapturing device.

FIG. 17 is a diagram illustrating a spectral transmittance of an RGB3-color optical filter.

FIG. 18 is a diagram illustrating color-matching functions of an XYZcolorimetric system regarding color perception of a human.

FIG. 19 is a diagram illustrating a spectral intensity distribution ofcertain colored light.

FIG. 20 is a diagram illustrating an example of spectral characteristicsof an infrared cut filter.

FIG. 21 is a diagram illustrating a configuration example of a colorimage input device.

FIG. 22 is a diagram illustrating another configuration example of thecolor image input device.

DESCRIPTION OF EMBODIMENTS First Example Embodiment

FIG. 1 is a block diagram illustrating a configuration of a videoprocessing device according to an example embodiment of the presentinvention. A video processing device 100 is a device that acquires avideo signal representing a video including near-infrared light, andoutputs a color signal and a near-infrared signal corresponding to theacquired video signal. In other words, the video processing device 100is a device that separates the color signal and the near-infrared signalfrom the video imaged in a state where the video includes visible lightand near-infrared light. Each arrow indicated in the subsequent blockdiagrams represent an example of a flow of a signal, but it is notintended that the flow of the signal is limited in a specific direction.

The term “video” described herein refers to an image to be capturedthrough an optical system, such as a lens, and may be either a stillimage or a moving image. The color signal is a signal representingvisible light components in the video signal. On the other hand, thenear-infrared signal is a signal representing near-infrared lightcomponents in the video signal. The color signal and the near-infraredsignal represent, for example, the brightness of a pixel, but are notlimited only to the brightness. In the following description, it isassumed that the color signal and the near-infrared signal representbrightness of each pixel of a still image or an image of a video at aspecific time of the video.

In this example embodiment, the term “visible range” refers to awavelength range of 400 to 700 nm. In the wavelength range, a blue (B)wavelength range is from 400 to 500 nm, a green (G) wavelength range isfrom 500 to 600 nm, and a red (R) wavelength range is from 600 to 700nm. The term “near-infrared range” refers to a wavelength range of 700nm to 2.5 μm. However, classification of the wavelength ranges indicatedherein is merely an example.

The video processing device 100 includes an acquisition unit 110 and asignal processing unit 120. The video processing device 100 is connectedto an external device that supplies a video signal. The external deviceis, for example, an imaging device including an image sensor. Theacquisition unit 110 acquires a video signal from the external device.The signal processing unit 120 outputs a color signal and anear-infrared signal based on the video signal acquired by theacquisition unit 110.

The acquisition unit 110 acquires a video signal representing a videoincluding near-infrared light having an intensity corresponding to apattern having a prescribed geometric shape. The term “pattern”described herein refers to a pattern in which figures, such as a circleor a rectangle, are repeatedly regularly arranged. Such a video signalis obtained in such a manner that, for example, the image sensor isprovided with an optical filter (hereinafter referred to as an “NIR cuttilter”) that cuts near-infrared light.

FIG. 2 is a diagram illustrating an NIR cut filter vertically withrespect to a light incident direction. An NIR cut filter 10 illustratedin FIG. 2 has a configuration in which the filter unit 11 is providedwith a plurality of infrared transmissive portions 12. In this case, theinfrared transmissive portions 12 are circular holes arranged at regularintervals. In the NIR cut filter 10, each infrared transmissive portion12 transmits near-infrared light, without blocking the near-infraredlight, and the filter unit 11 cuts a prescribed ratio or more of thenear-infrared light.

The pattern in which near-infrared light components included in thevideo signal are formed on the video does not necessarily match thepattern of the NIR cut filter. This is because the near-infrared lightcauses diffraction after the near-infrared light is transmitted throughthe NIR cut filter. Each figure in the pattern of the near-infraredlight components appearing as a video in the video signal is larger thanthe pattern (corresponding to the infrared transmissive portion 12illustrated in FIG. 2) formed in the NIR cut filter.

The signal processing unit 120 acquires such a video signal from theacquisition unit 110, and outputs a color signal and a near-infraredsignal. The color signal is typically output as signals of threecomponents of R, G, and B, but is not necessarily limited to this form.The signal processing unit 120 executes prescribed arithmetic processingby using pattern information for defining a pattern of the near-infraredlight, thereby enabling the color signal and the near-infrared signal tobe output.

The pattern information is information for specifying a pattern of thenear-infrared light on the video. For example, the pattern informationis data representing a position and shape of the pattern in an NIR cutfilter. In the case of the NIR cut filter illustrated in FIG. 2, thepattern information may be data in which the coordinates of the centerof a circle of each infrared transmissive portion 12 and the radiusthereof are described as parameters, but instead any data may be used aslong as the pattern can be specified. The pattern information may bepreliminarily stored in the video processing device 100, or may be inputby a user or the like. For example, the pattern information may bepreliminarily obtained in such a manner that a user carries out acalibration.

As described above, the video processing device 100 can separate thecolor signal and the near-infrared signal from the video signal by usinga combination of the video signal representing the video includingnear-infrared light with a prescribed pattern and the patterninformation. Therefore, according to the video processing device 100, itis possible to respectively execute visible light video processing andnear-infrared light video processing based on the video signal includingthe color signal and the near-infrared signal.

Further, when the video signal is obtained by an imaging device, it issufficient to use a typical imaging device provided with the NIR cutfilter as illustrated in FIG. 2, and thus a typical configuration can beused. In this example embodiment, the use of the output near-infraredsignal is not particularly limited.

Second Example Embodiment

FIG. 3 is a block diagram illustrating a configuration of a videoprocessing device according to another example embodiment of the presentinvention. A video processing device 200 illustrated in FIG. 3 includesa video data acquisition unit 210, a first color signal acquisition unit220, a pattern storage unit 230, a second color signal estimation unit240, a near-infrared signal calculation unit 250, and an output unit260. The video processing device 200 has functions similar to those ofthe video processing device 100 according to the first exampleembodiment.

The video data acquisition unit 210 acquires video data. The video dataacquisition unit 210 can acquire video data from an external devicesimilar to that of the first example embodiment. The video data includesat least a plurality of color signals. In this case, the plurality ofcolor signals are color signals represented by separate color componentsof three colors of R, G, and B, and each pixel is represented by a valueof a prescribed bit number. The term “color signals” described hereinrefers to a video in a state where near-infrared light components aresuperimposed on visible light components. Such color signals arehereinafter also referred to as a “first color signal”. The first colorsignal is a signal obtained by adding a second color signal and anear-infrared signal which are described below.

The first color signal acquisition unit 220 acquires the first colorsignal from the video data acquisition unit 210. The first color signalacquisition unit 220 acquires the first color signal for each color.

The pattern storage unit 230 stores pattern information. The patternstorage unit 230 is composed of, for example, a storage medium such as ahard disk or a flash memory. As the pattern information of this exampleembodiment, data similar to that of the first example embodiment can beused. As the pattern information, data common to each color can be used.

The second color signal estimation unit 240 estimates the second colorsignal which is a color signal obtained by removing near-infrared lightcomponents from the first color signal. Further, the second color signalestimation unit 240 estimates not only the second color signal, but alsothe intensity ratio between the second color signal and thenear-infrared signal. The second color signal estimation unit 240estimates the second color signal for each color and the intensity ratiothereof based on the first color signal acquired by the first colorsignal acquisition unit 220 and the pattern information stored in thepattern storage unit 230.

The near-infrared signal calculation unit 250 calculates a near-infraredsignal for each color. The near-infrared signal calculation unit 250 cancalculate a near-infrared signal by using the second color signalestimated by the second color signal estimation unit 240 and theintensity ratio between the second color signal and the near-infraredsignal.

The output unit 260 outputs a second color signal and a near-infraredsignal. The output unit 260 executes a prescribed calculation (e.g.,addition) on the near-infrared signal for each color calculated by thenear-infrared signal calculation unit 250, and outputs calculationresults.

The first color signal acquisition unit 220, the second color signalestimation unit 240, and the near-infrared signal calculation unit 250may sequentially execute processing for each color, or maysimultaneously execute the processing in parallel.

FIG. 4 is a block diagram illustrating the configuration of the secondcolor signal estimation unit 240 in more detail. The second color signalestimation unit 240 includes an initial value estimation unit 241, anestimated value selection unit 242, a smoothness evaluation unit 243, afirst color signal estimation unit 244, an error calculation unit 245,and an estimated value update unit 246.

The initial value estimation unit 241 calculates initial values ofestimated values of the second color signal and the intensity ratiobetween the second color signal and the near-infrared signal. Theinitial value estimation unit 241 calculates initial values of theestimated value of the second color signal and the estimated value ofthe intensity ratio based on the first color signal.

The estimated value selection unit 242 selects the estimated values ofthe second color signal and the intensity ratio. The estimated valueselection unit 242 repeatedly performs processing of selecting theseestimated values. In the initial selection processing, the estimatedvalue selection unit 242 selects the initial value calculated by theinitial value estimation unit 241, while in the second and subsequentselection processing, the estimated value selection unit 242 selects anestimated value updated by the estimated value update unit 246.

The smoothness evaluation unit 243 evaluates smoothness of the estimatedvalues of the second color signal and the intensity ratio. In this case,the smoothness indicates a degree of spatial variation in values. Forexample, the phrase “the estimated value of the second color signal issmooth” indicates that the difference between a maximum value and aminimum value of an estimated value of each pixel in a certain rangeconstituting a video is equal to or less than a prescribed threshold.The smoothness evaluation unit 243 calculates an evaluated value of thesmoothness according to a prescribed algorithm.

The first color signal estimation unit 244 estimates the first colorsignal. The first color signal estimation unit 244 calculates anestimated value of the first color signal based on the estimated valueselected by the estimated value selection unit 242 and the patterninformation stored in the pattern storage unit 230.

The error calculation unit 245 compares the estimated value of the firstcolor signal with the actual first color signal, and calculates an errortherebetween. Specifically, the error calculation unit 245 compares thefirst color signal estimated by the first color signal estimation unit244 with the first color signal acquired by the first color signalacquisition unit 220.

The estimated value update unit 246 updates the estimated values of thesecond color signal and the intensity ratio. The estimated value updateunit 246 updates the estimated values based on the estimated valuecalculated by the smoothness evaluation unit 243 and the errorcalculated by the error calculation unit 245.

Further, the estimated value update unit 246 compares the estimatedvalues before and after the update. When the amount of update of eachestimated value is sufficiently small, the update is finished.Specifically, the estimated value update unit 246 compares the amount ofupdate of each estimated value with a prescribed threshold, and when theamount of update is equal to or less than the threshold, the estimatedvalue update unit 246 finishes the update. The estimated value updateunit 246 sets the estimated value obtained at the time when the updateis finished as an output value of the second color signal estimationunit 240.

On the other hand, when the amount of update exceeds the threshold, theestimated value update unit 246 supplies the estimated value to theestimated value selection unit 242. In this case, the estimated valueselection unit 242, the smoothness evaluation unit 243, the first colorsignal estimation unit 244, the error calculation unit 245, and theestimated value update unit 246 executes the above-described processingby using the updated estimated value, and repeatedly perform thisprocessing until the update of the estimated value is finished.

The video processing device 200 has a configuration as described above.In this configuration, when the video processing device 200 acquiresvideo data, the video processing device 200 outputs the color signal andthe near-infrared signal. Specific operations of the video processingdevice 200 will be described below. It is assumed herein that colorsignals for R, G, and B colors are set to all pixels of video data.

FIG. 5 is a flowchart schematically illustrating processing executed bythe video processing device 200. However, the video processing device200 need not necessarily execute the processing as illustrated in FIG.5. For example, the video processing device 200 may execute processingof steps S3 and S4 in parallel on the color signals for R, G, and Bcolors.

First, the video data acquisition unit 210 acquires video data (stepS1). Next, the first color signal acquisition unit 220 selects any oneof a plurality of first color signals included in the video dataacquired by the video data acquisition unit 210 (step S2). At this time,the first color signal acquisition unit 220 selects a first color signalon which the processing of steps S3 and S4, which are described below,has not been executed yet.

When any one of the first color signals is selected by the first colorsignal acquisition unit 220, the second color signal estimation unit 240estimates a second color signal and an intensity ratio between thesecond color signal and a near-infrared signal based on the selectedfirst color signal (step S3). In other words, the second color signalestimation unit 240 calculates an estimated value of the second colorsignal and an estimated value of the intensity ratio. Next, thenear-infrared signal calculation unit 250 calculates a near-infraredsignal based on these estimated values (step S4).

After obtaining the necessary second color signal and near-infraredsignal, the output unit 260 outputs the second color signal andnear-infrared signal. Specifically, the output unit 260 determineswhether or not the processing of steps S2 to S4 has been executed forall colors (step S5). When the processing for all colors has beenfinished (step S5: YES), the second color signal and the near-infraredsignal are output (step S6).

On the other hand, when there is any color for which the processing ofsteps S2 to S4 has not been executed (step S5: NO), the first colorsignal acquisition unit 220 selects a first color signal that has notbeen processed (step S2). The second color signal estimation unit 240and the near-infrared signal calculation unit 250 executes theprocessing of steps S3 and S4 again according to the selection in stepS2.

The processing of steps S3 and S4 will be described in more detailbelow. The following description is made by using a color of “G” forconvenience of explanation, but processing for other colors is performedin a similar manner.

FIG. 6 is a diagram illustrating the near-infrared signal according tothis example embodiment, and illustrates a near-infrared signal that istransmitted through a circular infrared transmissive portion provided inthe NIR cut filter. In this case, an X-axis and a Y-axis respectivelycorrespond to Cartesian coordinates defined in a video represented byvideo data. A Z-axis represents brightness of the near-infrared signal.

The near-infrared signal has a value significant to a range wider thanthe actual area of the infrared transmissive portion due to an effect ofdiffraction of near-infrared light, and the value gradually decreasestoward the outside from the center of the infrared transmissive portion.When a distance between adjacent infrared transmissive portions isshort, the near-infrared signal may contain a mixture of componentsderived from a certain infrared transmissive portion and componentsderived from another infrared transmissive portion.

FIGS. 7A and 7B are diagrams each illustrating near-infrared lightcorresponding to the pattern of the NIR cut filter, and illustrate arange of near-infrared light irradiated on the image sensor. FIG. 7Aillustrates a case where infrared light beams from the respectiveinfrared transmissive portions do not overlap each other. On the otherhand, FIG. 7B illustrates a case where infrared light beams from therespective infrared transmissive portions overlap each other.

In this case, an intensity I(w) on the image sensor when thenear-infrared light having a wavelength λ and an incident intensity I₀is incident into one infrared transmissive portion located on the NIRcut filter is represented by the following Formula (1).

$\begin{matrix}{{I(w)} = {I_{0}\left\lbrack \frac{2{J_{1}(w)}}{w} \right\rbrack}^{2C}} & (1)\end{matrix}$

where J₁(w) represents the Bessel function of the first kind of order 1,and C represents a prescribed correction coefficient. The correctioncoefficient C is a coefficient for adjusting the intensity I(w) to matchthe pattern formed in actual video. “w” is represented by the followingFormula (2).

$\begin{matrix}{w = {\frac{2\; \pi \; a}{\lambda}\frac{q}{R}}} & (2)\end{matrix}$

where “a” represents a radius of the infrared transmissive portion. “q”and “R” respectively correspond to a distance between a point “p” and apoint where a perpendicular to the image sensor from the center of theinfrared transmissive portion intersects with the image sensor, and adistance between the center of the infrared transmissive portion and thepoint “p”, when any point on the image sensor is set as the point “p”.FIG. 8 illustrates “a”, “q”, and “R” in Formula (2).

FIG. 9 is a diagram illustrating a relationship between the distancefrom the center and the intensity of one pattern formed on the imagesensor when near-infrared light is diffracted by the NIR cut filter. Thecorrection coefficient C in Formula (1) is determined so that theintensity I(w) matches the pattern.

Accordingly, assuming that an intensity of the near-infrared signal at aposition X on the image corresponding to the unit that transmitsnear-infrared light (i.e., the infrared transmissive portion) isrepresented by I_(NIR) _(_) _(G)(X), an intensity I_(NIR) _(_) _(G)(X,x)that is observed in a pixel located at a position x by the near-infraredlight transmitted through the same infrared transmissive portion isrepresented by the following Formula (3).

I _(NIR) _(_) _(G)(X, x)=k _(X→x) I _(NIR) _(_) _(G)(X)   (3)

where k_(X→x) represents a coefficient calculated by using Formulas (1)and (2) from a distance between the position X and the position x on theimage sensor. However, the method for calculating the coefficientk_(X→x) is not limited to this method. As the method for calculating thecoefficient k_(X→x), for example, when spectral distribution of thenear-infrared signal I_(NIR) _(_) _(G)(X) is known, there is a methodfor combining coefficients calculated by using Formulas (1) and (2) ateach wavelength based on the spectral distribution. The coefficientk_(X→x) can also be obtained by calculation based on the standardspectral distribution of near-infrared light in a capturing scene, orcomputation in advance using another means.

Further, light that reaches the pixel located at the position x is mixedlight of visible light and near-infrared light. Specifically, the firstcolor signal in the pixel located at the position x is obtained byadding the second color signal, which is the color signal composed onlyof visible light components, and a signal representing near-infraredlight that reaches the pixel from a plurality of infrared transmissiveportions constituting the pattern on the NIR cut filter. Accordingly,the first color signal can be represented by I_(G) _(_) _(NIR)(x)illustrated in Formula (4). In the formula, I_(G)(x) corresponds to thesecond color signal representing a G component in the visible lightcomponents.

$\begin{matrix}{{I_{G\_ NIR}(x)} = {{I_{G}(x)} + {\sum\limits_{\forall X}{k_{X\rightarrow x}{I_{NIR\_ G}(X)}}}}} & (4)\end{matrix}$

where the relationship between the second color signal I_(G)(X) and thenear-infrared signal I_(NIR) _(_) _(G)(X) is represented by Formula (5)when using an intensity ratio m_(G). Formula (4) can be transformed intoFormula (6) by using Formula (5).

$\begin{matrix}{{I_{NIR\_ G}(X)} = {{m_{G}(X)}{I_{G}(X)}}} & (5) \\{{I_{G\_ NIR}(x)} = {{I_{G}(x)} + {\sum\limits_{\forall X}{k_{X\rightarrow x}{m_{G}(X)}{I_{G}(X)}}}}} & (6)\end{matrix}$

The use of the model formula represented by Formula (6) enablesestimation of the second color signal and the intensity ratio from thefirst color signal. Assuming that a vector including the first colorsignal (I_(G) _(_) _(NIR)(x)) for each pixel as an element isrepresented by I_(G) _(_) _(NIR), I_(G) _(_) _(NIR) can be theoreticallyrepresented by Formula (7).

I _(G) _(_) _(NIR) =I _(G) +KD(M _(G))SI _(G)   (7)

where I_(G) represents a vector including the second color signal(I_(G)(x)) for each pixel as an element; S represents a sampling matrixfor extracting the second color signal in a portion that transmits thenear-infrared light; D(M_(G)) represents a diagonal matrix including, asa diagonal element, each element of a vector M_(G) including the valueof an intensity ratio (m_(G)) for each portion that transmits thenear-infrared light as an element; and K represents a matrix includingthe value of the coefficient k_(X→x) as an element.

The second color signal I_(G) and the intensity ratio M_(G) are obtainedby calculating a value for minimizing an energy function E representedby the following Formula (8).

$\begin{matrix}{{E\left( {I_{G},M_{G}} \right)} = {{\frac{1}{2}{{I_{G\_ NIR} - \left( {I_{G} + {{{KD}\left( M_{G} \right)}{SI}_{G}}} \right)}}^{2}} + {\lambda_{1}{C_{1}\left( I_{G} \right)}} + {\lambda_{2}{C_{2}\left( M_{G} \right)}}}} & (8)\end{matrix}$

The first term on the right side of Formula (8) represents a valuelarger than 0 unless the second color signal I_(G) and the intensityratio M_(G) satisfy the relationship of Formula (7). The second andthird terms on the right side of Formula (8) are regularization termsfor preventing the energy minimization in the first term on the rightside derived from Formula (7) from being ill-posed. These terms areterms obtained by multiplying coefficients λ₁ and λ2 that arepreliminarily set to cost functions C₁(I_(G)) and C₂(M_(G)) forevaluating spatial smoothness of the second color signal and theintensity ratio.

The cost functions for evaluating the smoothness are represented by, forexample, C(p) in Formulas (9) and (10). In this case, Ω_(p) represents aset of pixels constituting the image sensor, and N(x) represents afunction indicating a position of a spatially adjacent pixel group.Further, p(x) represents data (the second color signal I_(G) or theintensity ratio M_(G)) corresponding to the pixel located at theposition x on the image sensor, and p(y) represents data (the secondcolor signal I_(G) or the intensity ratio M_(G)) corresponding to anyarbitrary pixel in a pixel group spatially adjacent to the pixel locatedat the position x.

$\begin{matrix}{{C(p)} = {\sum\limits_{x \in \Omega_{p}}{\sum\limits_{y \in {N{(x)}}}{{{p(x)} - {p(y)}}}}}} & (9) \\{{C(p)} = {\sum\limits_{x \in \Omega_{p}}{\sum\limits_{y \in {N{(x)}}}{{{p(x)} - {p(y)}}}^{2}}}} & (10)\end{matrix}$

The second color signal I_(G) and the intensity ratio M_(G) arecalculated in such a manner that, specifically, the second color signalestimation unit 240 updates the value by using a repeat operation. Thisrepeat operation is a repeat operation in which I⁰ _(G) is set as I_(G)NIR and values of all elements of M⁰ _(G) are set as a matrix of 1.0,and the update formulas as represented by Formulas (11) and (12) arerepeated until the amount of update becomes sufficiently small.

$\begin{matrix}{I_{G}^{t + 1} = {I_{G}^{t} + {k^{t}\left( {{\left( {E + {{{KD}\left( M_{G}^{t} \right)}S}} \right)^{T}\left( {I_{G\_ NIR} - {\left( {E + {{{KD}\left( M_{G}^{t} \right)}S}} \right)I_{G}^{t}}} \right)} - {\lambda_{1}\frac{\partial{C_{1}\left( I_{G}^{t} \right)}}{\partial I_{G}^{t}}}} \right)}}} & (11) \\{M_{G}^{t + 1} = {M_{G}^{t} + {k^{t}\left( {{\left( {{KD}\left( {SI}_{G}^{t + 1} \right)} \right)^{T}\left( {I_{G\_ NIR} - I_{G}^{t + 1} - {\left( {{KD}\left( {SI}_{G}^{t + 1} \right)} \right)M_{G}^{t}}} \right)} - {\lambda_{2}\frac{\partial{C_{2}\left( M_{G}^{t} \right)}}{\partial M_{G}^{t}}}} \right)}}} & (12)\end{matrix}$

where V^(t) _(G) and M^(t) _(G) respectively represent I_(G) and M_(G)at a repetition number t. k^(t) represents a coefficient for adjustingthe amount of updateat the repetition number t, and satisfies 0<k^(t)<1;E represents a unit matrix; and superscript T represents thetransposition of a matrix.

Thus, when the second color signal I_(G) and the intensity ratio M_(G)are calculated, the near-infrared signal I_(NIR) _(_) _(G) can becalculated. Specifically, when the near-infrared signal calculation unit250 substitutes the second color signal I_(G) and the intensity ratioM_(G) into Formula (5), thereby calculating a near-infrared signalI_(NIR G).

The second color signal and the near-infrared signal for R and Bcomponents can also be calculated in a manner similar to those for the Gcomponent. Specifically, by a calculation similar to that for the Gcomponent, the second color signal I_(R) and the near-infrared signalI_(NIR) _(_) _(R) for the R component, and the second color signal I_(B)and the near-infrared signal I_(NIR) _(_) _(B) for the B component canbe calculated.

A near-infrared signal output from the video processing device 200 isobtained by adding the near-infrared signals corresponding to the R, G,and B components, respectively. Specifically, assuming that the outputnear-infrared signal is represented by I_(NIR), I_(NIR) is representedby the following Formula (13).

I _(NIR) =I _(NIR) _(_) _(R) +I _(NIR) _(_) _(G) +I _(NIR) _(_) _(B)  (13)

The video processing device 200 executes such arithmetic processing,thereby outputs video data including the near-infrared signal I_(NIR)and second color signals I_(R), I_(G), and I_(B). The video processingdevice 200 can obtain the second color signal and the near-infraredsignal from the first color signal only by preparing an NIR cut filterand pattern information corresponding to the NIR cut filter. In thiscase, the imaging device does not require any special configurationother than the NIR cut filter.

Third Example Embodiment

FIG. 10 is a schematic diagram illustrating a configuration of acapturing device according to further another example embodiment of thepresent invention. A capturing device 300 illustrated in FIG. 10includes a light receiving unit 310 and a video processing unit 320.More specifically, the light receiving unit 310 includes an NIR cutfilter 311, a color filter 312, and a photo sensor 313. Light includingvisible light and near-infrared light is incident on the capturingdevice 300 through an optical system such as a lens.

The NIR cut filter 311 is an optical filter having a configurationsimilar to that of the NIR cut filter according to the first and secondexample embodiments. The NIR cut filter 311 is provided on the frontside in a travelling direction of the incident light with respect to thecolor filter 312 and the photo sensor 313. The NIR cut filter 311 isprovided at a prescribed distance from the color filter 312 and thephoto sensor 313 so that the near-infrared light diffused by diffractionis received by the photo sensor 313. The NIR cut filter 311 may bedetachably or movably configured.

FIG. 11 is a schematic view representing a behavior of near-infraredlight incident on the light receiving unit 310. As illustrated in FIG.11, the near-infrared light is transmitted through a part (infraredtransmissive portion) of the NIR cut filter 311, but is cut in the otherportions. However, the near-infrared light is diffracted when passingthrough the infrared transmissive portion, and thus the near-infraredlight is incident in a range wider than the infrared transmissiveportion in the photo sensor 313.

The color filter 312 is a three-color optical filter having a typicalconfiguration. The color filter 312 has, for example, spectralcharacteristics as illustrated in FIG. 17. The photo sensor 313 includesa plurality of photoelectric elements (i.e., sensors) to generate asignal corresponding to the intensity of incident light. The photosensor 313 may have a configuration similar to that of a typical imageinput device or capturing device. The video processing unit 320 acquiresthe signal generated by the photo sensor 313, and executes videoprocessing. The video processing unit 320 has the same functions asthose of the video processing device 200 of the second exampleembodiment, and also has a function for executing demosaicing processingas described below.

FIG. 12 is a diagram illustrating a part of the configuration of thecolor filter 312. As illustrated in FIG. 12, the color filter 312 is aso-called Bayer type array. In the color filter 312, filters areprovided so as to correspond to the respective sensors (i.e., pixels) ofthe photo sensor 313.

The pattern of the NIR cut filter 311 may have a correspondence relationwith an array of pixels of the photo sensor 313. The term“correspondence relation” described herein indicates that, for example,an interval between infrared transmissive portions of the NIR cut filter311 is equal to an interval between the pixels of the photo sensor 313,or has an integral multiple relationship. Specifically, the infraredtransmissive portions of the NIR cut filter 311 may be provided so as tooverlap the pixels corresponding to a specific color in the photo sensor313. However, the pattern of the NIR cut filter 311 need not necessarilyhave the correspondence relation with the array of pixels of the photosensor 313.

The capturing device 300 has a configuration as described above. Thecapturing device 300 can generate video data represented by four colors(four components) of R, G, B, and NIR based on video data represented bythree colors of R, G, and B in this configuration. Major points ofoperations of the capturing device 300 are the same as those of thevideo processing device 200 of the second example embodiment. However,the capturing device 300 executes demosaicing processing prior to theoperations described in the second example embodiment.

FIG. 13 is a diagram for explaining an example of demosaicingprocessing, and illustrates the correspondence relation between pixelsand coordinates. In this case, for convenience of explanation,coordinates of (1,1), (1,2), (2,1), and (2,2) are respectively allocatedto two rows and two columns of pixels illustrated in FIG. 13. The pixelat the coordinates (1,1) corresponds to an R component. The pixel at thecoordinates (2,2) corresponds to a B component. The remaining pixelscorrespond to G components.

In the following description, pieces of color information (color signalvalues) each representing RGB colors at coordinates (i,j) arerespectively represented by R(i,j), G(i,j), and B(i,j). For example,R(1,1) represents color information about the R component of the pixelat the coordinates (1,1). The color information obtained at the timewhen demosaicing processing is executed actually includes NIRcomponents. However, it is assumed herein that, for convenience ofexplanation, the NIR components of the color information are not takeninto consideration.

The pixel at the coordinates (1,1) corresponds to the R component.Accordingly, the color information about the R component at thecoordinates (1,1) is represented by the following Formula (14).

R(1,1)=R(1,1)   (14)

On the other hand, the pixel at the coordinates (1,1) does not receiveother color components. Accordingly, the color information about the Gand B components of the pixel at the coordinates (1,1) is obtained byinterpolating peripheral pixels as expressed by Formulas (15) and (16).

G(1,1)=(G(2,1)+G(1,2))/2   (15)

B(1,1)=B(2,2)   (16)

Next, the color information about the pixel at the coordinates (1,2) isexpressed by Formulas (17) to (19).

G(1,2)=G(1,2)   (17)

R(1,2)=R(1,1)   (18)

B(1,2)=B(2,2)   (19)

Note that the color information about the pixel at the coordinates (2,1)is obtained in a manner similar to that for the color information aboutthe pixel at the coordinates (1,2). Further, the color information aboutthe pixel at the coordinates (2,2) is obtained in a manner similar tothat for the color information about the pixel at the coordinates (1,1).

The video processing unit 320 executes such processing on all pixels toacquire color information for each color. Subsequently, the videoprocessing unit 320 calculates the near-infrared signal by theoperations described in the second example embodiment. The demosaicingprocessing is not limited to the method described above, but instead maybe executed by using, for example, methods disclosed in NPLs 4 to 6.

The capturing device 300 can provide advantageous effects similar tothose of the video processing device 200 according to the second exampleembodiment. Further, the capturing device 300 can disperse infraredlight by diffraction in the NIR cut filter 311. With this configuration,even when infrared light having an intensity at which the near-infraredsignal is saturated is incident on the photo sensor 313, the capturingdevice 300 can decrease the intensity of the near-infrared signal perpixel and can increase an apparent dynamic range.

The video processing unit 320 may neglect the near-infrared lightcomponents included in the color signals of R and B components obtainedafter the demosaicing processing. Specifically, as for the R and Bcomponents, the video processing unit 320 may regard the second term onthe right side of Formula (7) as 0 (i.e., I_(R) _(_) _(NIR)=I_(R), I_(B)_(_) _(NIR)=I_(B)). In this case, since I_(NIR) _(_) _(R)=I_(NIR) _(_)_(B)=0 holds, I_(NIR)=I_(NIR) _(_) _(G) is established by Formula (13).

FIG. 14 is a diagram illustrating a preferred correspondence relationbetween an infrared transmissive portion 311 a of the NIR cut filter 311and the color filter 312. FIG. 14 illustrates a positional relationshipbetween the NIR cut filter 311 and the color filter 312 as viewed alongthe light incident direction. All of the infrared transmissive portions311 a illustrated in FIG. 14 are located at a position where theinfrared transmissive portion 311 a overlaps the pixel corresponding tothe G component. When the NIR cut filter 311 has such a pattern, theeffect of an error caused when I_(NIR) _(_) _(R) and I_(NIR) _(_) _(B)are neglected can be reduced as compared with a case where the NIR cutfilter 311 does not have such a pattern (e.g., when the infraredtransmissive portion 311 a overlaps the pixel corresponding to the Rcomponent or the B component).

MODIFIED EXAMPLES

Example embodiments of the present invention are not limited to thefirst to third example embodiments described above. For example, thepresent invention can also be implemented by aspects of modifiedexamples described below. The present invention may also be implementedby aspects in which the first to third example embodiments and themodified examples are combined as appropriate.

(1) Modified Example 1

In the example embodiments of the present invention, the specific shapeof the pattern of the NIR cut filter is not limited, as long as thepattern can be described as pattern information. For example, in thepattern formed in the NIR cut filter, the shape of each infraredtransmissive portion is not limited to a circular shape, and all theinfrared transmissive portions need not necessarily have the same shape.

(2) Modified Example 2

In the example embodiments of the present invention, the visible lightcomponents are not limited to three components of R, G, and B. As thevisible light components, for example, three components of cyan (C),magenta (M), and yellow (Y) may be used. Further, the visible lightcomponents are not necessarily composed of three components, but insteadmay be composed of components less or more than three components.

(3) Modified Example 3

FIGS. 15 and 16 are diagrams each illustrating another example of thecapturing device. FIG. 15 is a diagram illustrating a capturing device400 having a so-called three-plate type configuration, i.e., aconfiguration in which sensors respectively corresponding to R, G, and Bcolors are provided separately. FIG. 16 is a diagram illustrating acapturing device 500 including a so-called stacked sensor. The presentinvention can also be applied to a capturing device having such aconfiguration.

The capturing device 400 includes a prism 410, photo sensors 420, 430,and 440, an NIR cut filter 450, and a video processing unit 460. Theprism 410 decomposes incident light and outputs the decomposed light indirections corresponding to the R, G, and B components, respectively.The photo sensors 420(R), 430(G), and 440(B) each generate a signalcorresponding to the intensity of incident light of each color.

The NIR cut filter 450 is an optical filter similar to the NIR cutfilter 311 of the third example embodiment. There is no need to providethe NIR cut filter 450 in all the photo sensors 420, 430, and 440. TheNIR cut filter 450 may be provided in any one of the photosensors(photosensor 420 in FIG. 15) depending on the spectral characteristicsof the prism 410. In the case of the example illustrated in FIG. 15, itis assumed that the near-infrared light incident on the photo sensors430 and 440 is sufficiently less than the near-infrared light incidenton the photo sensor 420. For example, an optical filter for cuttingnear-infrared light (however, unlike the NIR cut filter 450, the opticalfilter has no pattern for transmitting near-infrared light formed,) maybe provided at a pre-stage of each of the photo sensors 430 and 440.

The video processing unit 460 may have a configuration similar to thatof the video processing unit 320 described in the third exampleembodiment. However, in the example illustrated in FIG. 15, only thecolor signal corresponding to the R component includes near-infraredlight components. Accordingly, the video processing unit 460 may executeprocessing for separating the near-infrared signal from the color signalonly on the color signal corresponding to the R component.

The capturing device 500 includes an NIR cut filter 510, a stackedsensor 520, and a video processing unit 530. The NIR cut filter 510 andthe video processing unit 530 may have configurations similar to thoseof the NIR cut filter 450 and the video processing unit 460,respectively, illustrated in FIG. 15.

The stacked sensor 520 is a sensor in which sensors 521, 522, and 523are stacked. The sensor 521 has sensitivity in the wavelength range ofthe B component. The sensor 522 has sensitivity in the wavelength rangeof the G component. The sensor 523 has sensitivity in thewavelengthranges of the R component and the near-infrared light components.

(4) Modified Example 4

The whole or a part of the configuration according to the presentinvention can be implemented by a computer. For example, the videoprocessing devices 100 and 200 and the video processing unit 320 can beimplemented by a processing device (processor), such as a centralprocessing unit (CPU), and a memory. The present invention may also beimplemented by a general-purpose processor or a processor dedicated tovideo processing.

The present invention may also be provided in the form of a program thatcan be executed by a computer. This program may be provided in the formin which the program is downloaded from another device (a server or thelike) via a network, or may be provided in the form of acomputer-readable storage medium. Furthermore, the present invention canbe provided not only as a video processing device, a capturing device, aprogram, and a storage medium, but also as a video processing method.

The present invention has been described above by citing the exampleembodiments described above as exemplary embodiments. However, thepresent invention is not limited to the example embodiments describedabove. In other words, the present invention can be applied in variousforms that can be understood by those skilled in the art within thescope of the present invention.

This application is based upon and claims the benefit of priority fromJapanese patent application No. 2015-184885, filed on Sep. 18, 2015, thedisclosure of which is incorporated herein in its entirety by reference.

REFERENCE SIGNS LIST

-   10, 311 NIR cut filter-   12, 311 a Infrared transmissive portion-   100, 200 Video processing device-   110 Acquisition unit-   120 Signal processing unit-   210 Video data acquisition unit-   220 First color signal acquisition unit-   230 Pattern storage unit-   240 Second color signal estimation unit-   241 Initial value estimation unit-   242 Estimated value selection unit-   243 Smoothness evaluation unit-   244 First color signal estimation unit-   245 Error calculation unit-   246 Estimated value update unit-   250 Near-infrared signal calculation unit-   260 Output unit-   300, 400, 500 Capturing device-   310 Light receiving unit-   312 Color filter-   313 Photo sensor-   320 Video processing unit

What is claimed is:
 1. An image processing device comprising: anacquisition unit configured to acquire a video signal representing avideo including near-infrared light having an intensity corresponding toa pattern having a prescribed geometric shape; and a signal processingunit configured to output a color signal and a near-infrared signal byusing pattern information for defining the pattern, the color signalrepresenting a visible light component corresponding to the acquiredvideo signal, the near-infrared signal representing a near-infraredlight component corresponding to the video signal.
 2. The imageprocessing device according to claim 1, wherein the pattern informationindicates a position and a shape of the pattern.
 3. The image processingdevice according to claim 1, wherein, when a difference between theacquired video signal and an estimated value of the video signalcalculated by using an estimated value of the color signal, an estimatedvalue of an intensity ratio between the color signal and thenear-infrared signal, and the pattern information satisfies a firstcondition, and a spatial variation in an estimated value of the colorsignal and an estimated value of the intensity ratio satisfies a secondcondition, the signal processing unit outputs, as an output signal, anestimated value of the color signal and a value of the near-infraredsignal obtained from an estimated value of the color signal and anestimated value of the intensity ratio.
 4. The image processing deviceaccording to claim 1, further comprising a light receiving unitincluding an optical filter including a transmissive unit configured totransmit near-infrared light with the pattern.
 5. The image processingdevice according to claim 4, wherein the light receiving unit includes aplurality of sensors each corresponding to one of a plurality of colorcomponents, and the pattern has a correspondence relation with an arrayof the plurality of sensors.
 6. (canceled)
 7. An image processing methodcomprising: acquiring a video signal representing a video includingnear-infrared light having an intensity corresponding to a patternhaving a prescribed geometric shape; and outputting, by using patterninformation for defining the pattern, a color signal representing avisible light component corresponding to the acquired video signal and anear-infrared signal representing a near-infrared light componentcorresponding to the video signal.
 8. A computer-readable programrecording medium recording a program causing a computer to execute:processing of acquiring a video signal representing a video includingnear-infrared light having an intensity corresponding to a patternhaving a prescribed geometric shape; and processing of outputting, byusing pattern information for defining the pattern, a color signalrepresenting a visible light component corresponding to the acquiredvideo signal and a near-infrared signal representing a near-infraredlight component corresponding to the video signal.